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(54) Color interpolation for image sensors using a local linear regression method 



(57) An image processing system and demosaicing 
method are provided to calculate estimated missing 
color sensor values (125) in an image using a linear pre- 
diction from the raw color sensor value (30) at the cur- 
rent pixel location. The raw image is divided into regions 
of sensor values (30), and the linear relations between 



color planes for each region are determined by a regres- 
sion method that calculates the degree to which different 
color planes co-vary within each region. The missing 
color sensor values (125) per region are calculated as 
a scaled and shifted version of the raw color sensor val- 
ues (30) using linear regression coefficients (115) deter- 
mined from the local linear regression process. 
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Description 

BACKGROUND OF THE INVENTION 

Technical Field of the Invention 

[0001] The present invention relates generally to im- 
age sensors, and specifically to image processing of 
sensor values. 

Description of Related Art 

[0002] Electronic image sensors are predominately of 
two types: CCDs (Charge Coupled Devices) and CMOS 
- APS (Complimentary Metal Oxide Semiconductor - Ac- 
tive Pixel Sensors). Both types of sensors typically con- 
tain an array of photo-detectors, arranged in a pattern, 
that sample light and/or color within an image. Each 
photo-detector corresponds to a pixel of an image and 
measures the intensity of light of the pixel within one or 
more ranges of wavelengths. 

[0003] In addition, both types of sensors may include 
a color filter array (CFA), such as the CFA described in 
U.S. Patent No. 3,971, 065 to Bayer (hereinafter referred 
to as Bayer), which is hereby incorporated by reference. 
With the Bayer CFA, each pixel sees only one wave- 
length range, corresponding to the perceived color red, 
green or blue. The Bayer mosaic pattern of color filters 
is shown below (the letters R, G1 , G2, and B represent 
the colors red, green on the red rows, green on the blue 
rows, and blue, respectively, for a single pixel). 
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[0004] To obtain the sensor values for all three prima- 
ry colors at a single pixel location, it is necessary to in- 
terpolate the color sensor values from adjacent pixels. 
This process of interpolation is called demosaicing. 
There are a number of demosaicing methods known to- 
day. Byway of example, but not limitation, various dem- 
osaicing methods have included pixel replication, bilin- 
ear interpolation and median interpolation. 
[0005] Many of the existing demosaicing algorithms 
interpolate the missing sensor values from neighboring 
sensor values of the same color plane, under the as- 
sumption that the sensor values of neighboring pixels 
are highly correlated in an image (hereinafter referred 
to as the neighboring correlation assumption). However, 
for image regions with sharp lines and edges, the cor- 
relation among neighboring pixels may be poor. There- 
fore, demosaicing based on the neighboring correlation 
assumption may generate color aliasing artifacts along 
edges and in regions with fine details. In addition, neigh- 



boring correlation assumption demosaicing methods 
may generate images with independent noise levels 
among the color planes, resulting in higher noise ampli- 
fication during color correction processing. 

5 [0006] Other demosaicing algorithms have incorpo- 
rated both the neighboring sensor values and the raw 
sensor value of the current pixel when calculating the 
missing color values. Such algorithms operate under the 
assumption that different color sensor values of the 

10 same pixel are usually highly correlated (hereinafter re- 
ferred to as the color correlation assumption). The cor- 
relation among the different colors is assumed to either 
be fixed for all images or the same across a single im- 
age. Color correlation assumption demosaicing meth- 

15 ods can offer improved edge and line reconstruction 
with less chromatic aliasing. However, in some images, 
the improved edge and line reconstruction comes at the 
cost of reduced color saturation due to assumptions of 
fixed positive correlation among different color planes. 

20 Therefore, what is needed is a demosaicing algorithm 
that improves edge and line reconstruction in an image 
without reduced color saturation. In addition, what is 
needed is a demosaicing algorithm that is tolerant of 
noise amplification during the color correction process. 

25 

SUMMARY OF THE INVENTION 

[0007] Embodiments of the present invention provide 
an image processing system implementing a demosai- 

30 cing algorithm that calculates estimated missing color 
sensor values in an image using a linear prediction from 
the raw color sensor values. The raw image is divided 
into regions of sensor values, and the linear relations 
between color planes for each region are determined by 

35 a regression method that calculates regression coeffi- 
cients corresponding to the degree to which different 
color planes co-vary in a region. The missing color sen- 
sor values per region are calculated as a scaled and 
shifted version of the raw color sensor values, using the 

40 linear regression coefficients determined from the local 
linear regression process. 

[0008] In one embodiment, a simple demosaicing 
process, such as bilinear interpolation, is applied to a 
region of sensor values prior to determining the linear 
45 regression coefficients between different color planes 
for the region. In other embodiments, the regression co- 
efficients are determined from the raw sensor values 
themselves. 

[0009] In further embodiments, where the assumption 
50 of a single linear correlation is violated for image regions 
with multiple object colors, the missing sensor values 
can be interpolated using the simple demosaiced results 
previously calculated or using a more sophisticated lin- 
ear regression method to determine multiple regression 
55 coefficients for regions with several different linear rela- 
tionships. 

[0010] Since all missing pixels of a color channel with- 
in one region are calculated with a single set of linear 
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regression coefficients, the linear relations between 
sensor values of the pixels within the region are pre- 
served, resulting in less blurring and less chromatic 
aliasing in the final image compared with neighboring 
correlation assumption demosaicing methods. In addi- 5 
tion, the noise terms between different color channels 
are correlated, thereby reducing noise amplification in 
subsequent processing compared with other neighbor 
correlation and color correlation assumption demosaic- 
ing methods. Furthermore, the invention provides em- 10 
bodiments with other features and advantages in addi- 
tion to or in lieu of those discussed above. Many of these 
features and advantages are apparent from the descrip- 
tion below with reference to the following drawings. 

15 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0011] The disclosed invention will be described with 
reference to the accompanying drawings, which show 
important sample embodiments of the invention and 20 
which are incorporated in the specification hereof by ref- 
erence, wherein: 

FIG. 1 is a chart illustrating the linear correlation be- 
tween color planes of all pixels in an image; 25 
FIG. 2 is a chart illustrating the linear correlation be- 
tween color planes of pixels in different regions of 
an image; 

FIG. 3 is a block diagram illustrating an image 
processing system utilizing a local linear regression 30 
demosaicing algorithm in accordance with exem- 
plary embodiments of the present invention; 
FIG. 4 is a logic flow diagram illustrating exemplary 
logic for implementing the local linear regression 
demosaicing algorithm in accordance with exem- 35 
plary embodiments of the present invention; 
FIG. 5 is a flow chart illustrating exemplary steps for 
performing the local linear regression demosaicing 
algorithm illustrated in FIG. 4; 

FIG. 6 is a flow chart illustrating exemplary steps for 40 
performing the local linear regression demosaicing 
algorithm using raw and interpolated sensor values, 
in accordance with embodiments of the present in- 
vention; 

FIG. 7 is a flow chart illustrating exemplary steps for 45 
performing the local linear regression demosaicing 
algorithm using only raw sensor values, in accord- 
ance with other embodiments of the present inven- 
tion; 

FIG. 8 is a flow chart illustrating exemplary steps for 50 
performing the local linear regression demosaicing 
algorithm using a blurred image, in accordance with 
further embodiments of the present invention; 
FIG. 9 is a chart illustrating the violation of the linear 
correlation assumption in a particular region of an 55 
image; 

FIG. 10 is a logic flow diagram illustrating exemplary 
logic for implementing the linear regression demo- 



saicing algorithm with a linear correlation violation 
procedure in accordance with exemplary embodi- 
ments of the present invention; 
FIG. 11 is a flow chart illustrating exemplary steps 
for determining whether an image region violates 
the linear correlation assumption and replacing 
sensor values in violated areas, in accordance with 
embodiments of the present invention; and 
FIG. 12 is a flow chart illustrating exemplary steps 
for detecting edge regions within an image. 

DETAILED DESCRIPTION OF EXEMPLARY 
EMBODIMENTS 

[0012] The numerous innovative teachings of the 
present application will be described with particular ref- 
erence to exemplary embodiments. However, it should 
be understood that these embodiments provide only a 
few examples of the many advantageous uses of the 
innovative teachings herein. In general, statements 
made in the specification do not necessarily delimit any 
of the various claimed inventions. Moreover, some 
statements may apply to some inventive features, but 
not to others. 

[0013] For most image sensors, at any pixel location, 
the captured sensor values for different colors are highly 
correlated, meaning that the sensor values for different 
colors are predictably sloped and offset from one anoth- 
er. The correlation amongst the different colors is a re- 
sult of the photo-detectors at a pixel location having 
largely overlapping sensitivities and the fact that objects 
captured in an image generally have smooth surface re- 
flectance curves. 

[0014] However, the correlation amongst colors tends 
to be only moderate when calculated over an entire im- 
age that contains multiple objects of different colors. Re- 
ferring now to FIG. 1 , there is illustrated a chart depicting 
the linear correlation between two color planes for all 
pixels in a sample image (not shown). In FIG. 1, the 
green color value of a pixel location is plotted against 
the red colorvalueofthesame pixel location for all pixels 
in an image. As can be seen in FIG. 1 , there is an overall 
positive correlation between the red values and green 
values. The correlation coefficient is 0.71, which indi- 
cates that about 50% of the variance in green values 
can be predicted by the red values (the amount of vari- 
ance accounted for is calculated by the square of the 
correlation coefficient). 

[0015] However, there are subsets of pixels in FIG. 1 
that have different relations between red and green 
color values, shown as bands with different slopes in the 
scatter plot. To improve the correlation between red and 
green values for the pixels in each of these bands, a 
different linear relation between red and green values 
at pixel locations within the bands should be calculated 
for each of the bands. The correlation between red and 
green color values for the entire image (i.e., 0.71) is not 
high enough to produce accurate calculates of missing 
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pixel values at individual pixel locations. 
[0016] Referring now to FIG. 2, in accordance with 
embodiments of the present invention, the correlation 
levels improve significantly when the green values are 
plotted against the red values for smaller regions of the 
same image. In FIG. 2, five 8x8 regions of pixels have 
been randomly chosen from the image plotted in FIG. 
1 . Within four of the individual region, the correlation co- 
efficient is on average much higher (e.g., greater than 
0.9), although different regions have different slopes 
and offsets. For one of the regions, the correlation co- 
efficient is even negative. The higher correlation results 
in reduced color aliasing artifacts in images demosaiced 
based on the local correlation in regions of the image. 
[0017] Referring now to FIG. 3, there is illustrated an 
image device 5 having an image processing system 1 0 
implementing a local linear regression (LLR) demosai- 
cing algorithm 45, in accordance with the present inven- 
tion. The image device 5 can be incorporated as part of 
any digital imaging device, such as a camera, video 
camera, medical imaging device, etc. The image device 
5 can further be at least partially incorporated on a com- 
puter system, such as a personal computer or server, 
having a memory therein for storing image data. Thus, 
the algorithm 45 can be located within a digital imaging 
device or as part of an image processing software run- 
ning on a personal computer or server. 
[0018] The image device 5 includes an image sensor 
20, such as a CMOS sensor chip or a CCD sensor chip, 
which includes a two-dimensional array of pixels 25 ar- 
ranged in rows and columns; The image sensor 20 may 
be covered by a color filter array (CFA), such that each 
pixel 25 senses only one color. For example, the CFA 
can be the well-known Bayer CFA, in which chromi- 
nance colors (red and blue) are interspersed amongst 
a checkerboard pattern of luminance colors (green). It 
should be understood that the LLR demosaicing algo- 
rithm 45 described herein is also applicable to other CFA 
configurations, such as CMY (cyan, magenta, yellow) 
sensor mosaics, or other n-color (n33) sensor mosaics. 
[001 9] The image sensor 20 provides raw sensor val- 
ues 30 containing the original red, blue and green pixel 
values to a digital signal processor 40 within the image 
processing system 1 0 capable of applying the LLR dem- 
osaicing algorithm 45 of the present invention to the sen- 
sor values 30. The sensor values 30 are divided into re- 
gions and provided to the digital signal processor40 one 
region at a time. Thus, the sensor values 30 are stored 
in a buffer 50 until the requisite number of sensor values 
30 is present to begin processing. The buffer 50 can be 
implemented as a storage device external to the proc- 
essor 40 (e.g., RAM) or within the processor 40 (e.g., 
internal register, stack or cache). In other embodiments, 
the buffer 50 and/or processor 40 can be built into the 
sensor chip itself. 

[0020] The number of sensor values 30 needed to be- 
gin processing depends on the size of the regions that 
the image is divided into for processing purposes. For 



example, the sensor values 30 are typically read off the 
sensor 20 one row at a time. Therefore, to process an 
8x8 block of sensor values 30, eight rows of sensor 
values would need to be stored in the buffer 50. 
5 [0021] * The LLR demosaicing algorithm 45 deter- 
mines the linear relations between color planes for each 
region by a regression method that determines coeffi- 
cients of best-fitting linear functions that relate pixel val- 
ues of different color planes for each region. The miss- 
to ing color sensor values per region are calculated as a 
scaled and shifted version of the raw color sensor val- 
ues, using the linear regression coefficients estimated 
from the local linear regression process. 
[0022] After demosaicing of the image using the LLR 
15 demosaicing algorithm 45 is complete, the demosaiced 
(interpolated and raw) sensor values 130 can be used 
in subsequent processing. For example, the demo- 
saiced sensor values 130 can be subjected to a color 
correction process, compression process for storage 
20 purposes or color conversion process to output the im- 
age to an output device, such as a video screen, com- 
puter screen or printer. 

[0023] Referring now to FIG. 4, there is illustrated ex- 
emplary logic for implementing the LLR demosaicing al- 

25 gorithm 45, in accordance with embodiments of the 
present invention. The LLR demosaicing algorithm 45 
takes as input the raw sensor values 30 provided by the 
buffer 50. Interpolation logic 100 receives blocks of sen- 
sor values 30 at a time and calculates interpolated sen- 

30 sor values 35, using any fast and simple demosaicing 
process, such as bilinear or direct linear interpolation. 
The blocks correspond to the regions into which the im- 
age is divided. It should be understood that as used 
herein, the term "logic" refers to the hardware, software 

35 and/or firmware for performing the stated function. 

[0024] The raw sensor values 30, and in some em- 
bodiments (as shown in FIG. 6), the interpolated sensor 
values 35, for each region are provided to linear regres- 
sion logic 1 1 0 to determine linear regression coefficients 

40 115 (slope and intercept) between different color planes 
for each region. The linear regression coefficients 115 
are further provided to calculation logic 120 where they 
are used to calculate estimated missing color sensor 
values in the region from the raw sensor values using 

45 linear prediction. The calculated missing color sensor 
values are estimates of what the actual missing color 
sensor values would have been had the missing color 
sensor values been measured by the sensor. 
[0025] Since all missing color sensor values in a par- 

50 ticular region of the image are estimated from the raw 
color sensor values with the same linear regression co- 
efficients, the linear relations between sensor values of 
the pixels within the region are preserved, resulting in 
less blurring and less chromatic aliasing in the final dem- 

55 osaiced image. In addition, due to the fact that demo- 
saiced values and the raw values are linearly related, 
noise amplification in the color correction process that 
normally follows the demosaicing step can be reduced 
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using the LLR demosaicing algorithm of the present in- 
vention. The benefit of noise amplification reduction is 
especially significant when using image sensors that 
have broad-wavelength sensitivities (such as a CMY 
sensor). For these broad-wavelength sensors, the color 
correction matrix elements tend to have large values 
that contribute to noise amplification in the final proc- 
essed image. 

[0026] For example, assuming that the raw mosaiced 
image has an additive noise element n that is independ- 
ently and identically distributed for each pixel, with a 
mean of 0 and standard deviation of a, the color sensor 
values (e.g., red, green and blue) can be represented 
as: 

r= r 0 + n, 



9 = 9 0 + n, 
6 = b 0 + n, 

where r 0 , g 0 , b 0 are the true pixel values before any 
noise is added through the image capturing process. Af- 
ter traditional demosaicing by interpolation from neigh- 
boring pixels, the estimated color sensor value has a 
noise element that is correlated with the noise of its 
neighbors, but not with the noise element of the raw 
color sensor value of the same pixel (i.e., the noise el- 
ements are independent between the different color 
planes, and the overall noise level of the image is a com- 
bination of the independent noise elements of the three 
color channels). 

[0027] If (/' G {1 ,2,3}, j G {1 ,2,3}) is the color correc- 
tion matrix that converts from sensor RGB (or CMY) val- 
ues to display RGB values, the display RGB values, de- 
noted x n x g , x b , can be calculated as: 

x r =c 11 r+c 12 g+ c 13 b, 

*g= C 21 r+ C 220 + C 23 b > 



°* y = V(°21 + c 22 + c 23) a ' 



5 C x b = 7( C 31 + C 32 + C 33> a ' 

[0029] For an RGB sensor with a white-preserving 
color correction matrix (i.e., matrix rows sum to 1), the 
color matrix typically has diagonal terms that are greater 
10 than one and off-diagonal terms that are less than one 
or even negative. A color matrix example for an RGB 
sensor is shown in the array below. For such a matrix, 
the noise amplification is greater than one when demo- 
saiced r,g,b values have independent noise terms. 

15 



3.5973 


-1.4865 


-1.1109 


-0.4658 


1.9183 


-0.4526 


-0.1898 


-1.1079 


2.2977 



[0030] For CMY sensors, the color matrices tend to 
have much larger values, often with large negative 
terms, as shown in the example matrix below. The noise 
amplification is greater for these CMY sensors if the 
demosaiced r,g,b values have independent noise terms. 



3.5303 


2.4996 


-5.0298 


-1.8741 


1.1569 


1.7171 


2.2584 


-2.9010 


1.6426 



30 

[0031] With the LLR demosaicing algorithm of the 
present invention, missing color sensor values for all 
pixels in a local image region are calculated as a scaled 
and shifted version of the raw color sensor values. 
35 Therefore, the interpolated color sensor values corre- 
late well with the raw color sensor values in a local image 
region, and thus, the noise terms are highly correlated. 
The standard deviations of the noise terms for x n x gt 
x 6 become: 

40 

°~* r = ( C 11 + <M 2 + c 13 )a, 
4$ <*x g = ( C 21 + C 22 + C 23) a ' 



x b = c 31 r+ c 32 g + c 33 6. 

[0028] The noise distributions for x n x g , x b have 
means of 0s and standard deviations that are a function 
of Cy. When the demosaiced r,g,6 values have inde- 
pendent (or close to independent) noise elements, the 
standard deviations of the noise terms for x n x g ,x b are: 




°x b "( C 31 + C 32 + C 33)°~- 

so [0032] Since c n + c, 2 + ^3=1, the noise amplification 
factor in the local image region is one. Therefore, with 
the LLR demosaicing algorithm of the present invention, 
noise amplification is limited, which improves the final 
image quality. 

55 [0033] Exemplary steps within the LLR demosaicing 
algorithm are shown in FIG. 5. Initially, the measured 
raw digital sensor values representing an image are pro- 
vided to a digital signal processor (step 500) for demo- 
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saicing of the raw sensor values. These received raw 
sensor values are divided into regions for separate dem- 
osaicing (step 510). For example, the sensor values can 
be divided into n x n blocks ornx/n blocks, with each 
block containing the same or a different number of sen- 
sor values. Upon retrieving the sensor values from the 
buffer for a particular region, a linear regression of the 
raw sensor values in the region is performed to deter- 
mine the linear regression coefficients (slopes and off- 
sets) that describe the linear relationships between the 
different color planes in that region (step 520). From the 
linear regression coefficients, estimated missing color 
sensor values at each pixel location in that region are 
calculated (step 530). 

[0034] The linear regression coefficients are estimat- 
ed separately for each region using only the raw sensor 
values for that region. The missing color sensor values 
for each region are calculated separately using the spe- 
cific linear regression coefficients and raw sensor val- 
ues for that region. Once the missing color sensor val- 
ues in all regions of the image have been calculated 
(step 540), the final demosaiced image can be output 
from the digital signal processor for further processing 
or display (step 550). The final demosaiced image in- 
cludes both the original raw sensor values and the cal- 
culated missing color sensor values at each pixel loca- 
tion. 

[0035] In one embodiment, the linear regression co- 
efficients for each region are determined using both the 
raw sensor values and simple interpolated sensor val- 
ues. Turning now to FIG. 6, there is illustrated exemplary 
steps for performing the local linear regression demo- 
saicing algorithm using raw and interpolated sensor val- 
ues, in accordance with embodiments of the present in- 
vention. Upon receiving the raw sensor values (step 
600) and dividing the sensor values into regions (step 
610), as a first step, a simple demosaicing algorithm, 
such as bilinear interpolation or median interpolation, 
can be applied to the mosaiced image (step 620). The 
simple demosaicing process creates three full color 
planes (e.g., R, G, and B), each including both original 
raw and interpolated color sensor values. It should be 
understood that the simple demosaicing algorithm can 
be any simple and fast algorithm that does not alter the 
raw color sensor values. 

[0036] To determine the linear regression coefficients 
for each region, both the interpolated and raw color val- 
ues are used (step 630). For example, let r jt g jt 6, rep- 
resent individual r,g,b color sensor values for pixel / in a 
block of n x n pixels. To determine the regression coef- 
ficients (slopes S and offsets A) used to calculate R val- 
ues from G values (S^, A^), R values from B values 
(S,^ Afr), etc., variance and covariance terms must be 
calculated. For example, if r, g, b represent the mean 
•r,g,b values in the current image region, the sums of 
squares for the variance (SS) and covariance (CS) 
terms are: 
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55 



SS(R) = X;> f -r) 2 ; 



55(G) = j£(g t - g y 



ss(B) = £;> 2 -*) 2 ; 



CS(RG) = £;>,-r)0? ( -g); 



cs(GB) = E^CSi-fXA-*); 



cs(rb) = xlsn-n^-b). 

[0037] From the variance and covariance sums of 
squares, the linear regression coefficients Syy (slope) 
and Ayx (offset) that are used to predict color value y 
from color value x can be calculated as follows (only the 
coefficients for the R and G planes are shown for con- 
venience, although other color planes are similar): 



_ CS(RG) 
W SS(G) ' 



r-gs i 



rg 1 



SS(R) ' A 9r 9 r ^ 



gr 



[0038] When calculating the variance and covariance 
terms for a particular region, either all of the sensor val- 
ues for that particular region or only a subset of the sen- 
sor values from that region can be used, depending up- 
on the particular application. In other embodiments, the 
linear regression coefficients are not calculated, but 
rather look-up tables can be built to represent the linear 
functions relating different color planes based on aver- 
age intensities for the different color planes. 
[0039] Once the linear regression coefficients are de- 
termined, estimated missing color sensor values for the 
region can be calculated (step 640). For example, for all 
G pixel locations in the raw mosaic, the missing R and 
B values r,' g 1 can be calculated from the G values using 
Sfg, Afg and S bg> A bg in the following way: 



9b 



6 
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£>i = A bg + S b g* 9i- 

[0040] Similar estimations for the region can be ap- 
plied to ail of the missing R and G values at the B pixel 
locations, and all of the missing G and B values at the 
R pixel locations to result in a full three-color image for 
the region. This process is repeated for all regions (e. 
g., nxn blocks or n x m blocks) in the image (step 650) 
to produce a final demosaiced image (step 660). 
[0041] In other embodiments, the linear regression 
coefficients can be determined using only the raw sen- 
sor values. Turning now to FIG. 7, there is illustrated 
exemplary steps for performing the local linear regres- 
sion demosaicing algorithm using only raw sensor val- 
ues, in accordance with other embodiments of the 
present invention. Instead of performing a simple inter- 
polation step, upon receiving the raw sensor values 
(step 700) and dividing the sensor values into regions 
(step 710), the linear regression coefficients for each re- 
gion are determined from only the raw color sensor val- 
ues. 

[0042] For example, to determine the linear regres- 
sion coefficients for a Bayer mosaic image, let q repre- 
sent individual r color values for a particular image re- 
gion. For each r h let g /1f g^, g&, represent green pix- 
els that are immediately above, below, to the left, and to 
the right of r r As an initial step, correlation coefficients 
Pi« P2» P3« P4 between r and each of the four groups of 
g values are calculated (step 720) as follows: 



er color planes are similar): 



P/ = 



CS(RGj) 



JSS(Gj) x SS(R) ' 



where CS(RGfi and SS(Gy) are calculated in a similar 
fashion as shown above in connection with FIG. 6, ex- 
cept that the green values used are g ijf with / = 1 , 2, 3, 
4 corresponding to each of the green pixel positions. 
[0043] In a flat region of the image, the correlations 
between red and all four green values should be similar. 
However, in regions with a lot of high spatial frequency 
patterns, the py values can vary significantly. For exam- 
ple, in an image region with high horizontal frequency 
patterns, the correlation between rand g 3 , g 4 (left and 
right green pixels) will be high, whereas p 1 and p 2 will 
be low. Therefore, in order to avoid introducing color 
aliasing artifacts, the green pixel position that yields the 
highest correlation is selected to determine the linear 
regression coefficients between R and G color values 
(step 730). 

[0044] For example, if jmax denotes the green pixel 
position with the highest correlation with the red values 
for a particular region, the linear regression coefficients 
Syy (slope) and (offset) that are used in the region 
to predict color value y from color value x can be calcu- 
lated as follows (step 740) (only the coefficients for the 
r and g planes are shown for convenience, although oth- 



10 



CS{RG jmax ) 



rg 



SS(G ymax ) ' 



Afg ~ r ~ 9jmax^rg 



_ CS(RG jmax ) 

yr ss(R) ' 



■gr 



fymax " rS gr 



[0045] Once the linear regression coefficients for one 
set of color planes are determined, the process is re- 
peated for all other sets of color planes (step 750). From 
the linear regression coefficients for all of the color 

15 planes, estimated missing color sensor values for the 
region can be calculated (step 760) as described above 
in connection with FIG. 6. This process is repeated for 
all regions (e.g., nxn blocks or n x m blocks) in the 
image (step 770) to produce a final demosaiced image 

20 (step 780). 

[0046] In a further embodiment, to reduce the number 
of aliasing artifacts in the LLR demosaicing algorithm 
embodiment described above in connection with FIG. 6, 
the LLR demosaicing algorithm can be performed on on- 

25 |y the high spatial frequency components of the image. 
Referring now to FIG. 8, exemplary steps for performing 
the local linear regression demosaicing algorithm using 
the high spatial frequency components of the image are 
illustrated, in accordance with further embodiments of 

30 the present invention. 

[0047] Upon receiving the raw sensor values (step 
800) and dividing the sensor values into regions (step 
810), a simple demosaicing algorithm is applied to the 
mosaiced image to produce a first demosaiced version 

35 of the image region (step 820). Thereafter, a blurred ver- 
sion of the first demosaiced version of the image region 
is generated through a convolution with a gaussian ker- 
nel of a similar size as the region size (step 830). The 
gaussian convolution acts as a lowpass frequency filter, 

40 thereby removing the details from the image. It should 
be understood that other lowpass kernels can be used 
instead of the gaussian kernel. The blurred version is 
then subtracted from the first demosaiced version to 
generate a difference image, which contains only the 

45 high spatial frequency components of the original image 
(step 840). 

[0048] Subsequent regression and missing color sen- 
sor value estimation calculations are operated on the 
difference image (steps 850 and 860), as described 

50 above in connection with FIG. 6 or 7, to produce a sec- 
ond demosaiced version of the image region. After the 
second demosaiced version is completed for a region, 
the blurred version associated with the region is added 
back to the second demosaiced version (step 870) to 

55 produce a final demosaiced version of the image region. 
This process is repeated for all regions (e.g., nxn blocks 
or n x m blocks) in the image (step 880) to produce a 
final demosaiced image (step 890). 
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[0049] Although the general assumption of the 
present invention is that within a small local region, the 
R, G and B color sensor values have a high linear cor- 
relation, there may be some image regions where this 
assumption is violated. FIG. 9 shows blue color values 
plotted against corresponding green color values for all 
pixels in a particular region of a sample image. It can 
easily be seen in FIG. 9 that three linear functions are 
needed to describe the relationship between blue and 
green color values. Therefore, if only one linear function 
is used to predict blue from green, as is done in the local 
regression method described above, most blue sensor 
value calculates at the green pixel locations will be in- 
accurate, resulting in a grid-like error pattern. 
[0050] Grid artifacts typically involve groups of color 
sensor values that are all under calculated or over cal- 
culated, which results in color sensor values that are 
lower or higher than their neighboring pixels (hence the 
grid-like pattern). Therefore, to detect grid artifacts, a 
comparison can be made between the demosaicied 
color sensor values produced from the LLR demosaic- 
ing algorithm and the interpolated color values produced 
from the simple demosaicing step described above in 
connection with FIGs. 6 and 8. 

[0051] With reference now to FIG. 10, there is illus- 
trated exemplary logic for removing grid artifacts from 
an image demosaiced using the LLR demosaicing algo- 
rithm 45, in accordance with embodiments of the 
present invention. The LLR demosaicing algorithm 45 
takes as input the raw sensor values 30 provided by the 
buffer 50. Interpolation logic 100 receives blocks of sen- 
sor values 30 at a time and calculates interpolated sen- 
sor values 35, using any fast and simple demosaicing 
process, such as bilinear or direct linear interpolation. 
The blocks correspond to the regions into which the im- 
age is divided. 

[0052] The raw sensor values 30, and in some em- 
bodiments (as described above in connection with FIG. 
6), the interpolated sensor values 35, for each region 
are provided to linear regression logic 110 to determine 
the linear regression coefficients 115 (slope and inter- 
cept) between different color planes for each region. 
The linear regression coefficients 115 are further provid- 
ed to calculation logic 120 where they are used to cal- 
culate the estimated missing color sensor values in the 
region from the raw sensor values using linear predic- 
tion. 

[0053] The raw sensor values 30 are further provided 
to threshold logic 150 to compute a threshold amount 
1 55. In one embodiment, the threshold amount 1 55 can 
be variable depending on the light conditions of the im- 
age. For example, in low light conditions, the sensor val- 
ues are low and the signal to noise ratio is low, thus re- 
quiring a higher threshold amount 155 for determining 
whether a grid artifact has occurred. By contrast, in nor- 
mal or bright light conditions, the sensor values are high 
and the signal to noise ratio is high, thereby enabling a 
lower threshold amount 155 to be set for determining 



whether a grid artifact has occurred. In other embodi- 
ments, the threshold amount 1 55 can be preconfigured 
by the manufacturer or can be set by the camera oper- 
ator. 

5 [0054] The interpolated sensor values 35 and calcu- 
lated sensor values 1 25 are further provided to compar- 
ison logic 140 to determine whether any grid artifacts 
are present in the image region. Comparison logic 140 
compares the calculated sensor value 1 25 of each pixel 

10 to the corresponding interpolated sensor value 35 of 
each pixel and outputs difference values 145 for the 
color planes for each pixel. The difference values 145, 
the raw sensor values 30, the interpolated sensor values 
35, the calculated sensor values 125 and the threshold 

*5 amount 155 are provided to replacement logic 160 to 
output the final demosaiced sensor values 1 30. The final 
demosaiced sensor values 130 include the raw sensor 
values 30, the calculated sensor values 125 and the in- 
terpolated sensor values 35 as a substitute for the cal- 

20 culated sensor values 125 for those pixels where the 
difference value exceeding the threshold amount indi- 
cates that a grid artifact occurred as a result of LLR dem- 
osaicing. 

[0055] Exemplary steps for removing grid artifacts 

25 from an LLR demosaiced image are shown in FIG. 11. 
Upon receipt of the raw sensor values (step 900) and 
division of the sensor values into regions for separate 
demosaicing (step 905), a simple demosaicing algo- 
rithm is applied to the mosaiced image (step 910) to cre- 

30 ate three full color planes (e.g., R, G and B), each in- 
cluding both original raw and interpolated color sensor 
values. Thereafter, the linear regression coefficients 
(slopes and offsets) that describe the linear relation- 
ships between the different color planes in that region 

35 (step 915) are determined from both the interpolated 
and raw sensor values, as described above in connec- 
tion with FIG. 6, or from only the raw sensor values, as 
described above in connection with FIG. 7. From the lin- 
ear regression coefficients, estimated missing color 

40 sensor values at each pixel location in that region are 
calculated (step 920). 

[0056] For each calculated missing color sensor val- 
ue, a comparison is made between the calculated miss- 
ing color sensor value and the corresponding interpo- 

45 lated missing color sensor value to determine a differ- 
ence value (step 925). If the difference value exceeds a 
threshold amount (step 930), the calculated missing 
color sensor value is replaced with the interpolated 
missing color sensor value (step 935). Otherwise, the 

50 calculated missing color sensor value is used in the final 
demosaiced image (step 940). This process is repeated 
for each calculated missing color sensor value in the im- 
age region (step 945). 

[0057] Once the estimated missing color values in all 
55 regions of the image have been calculated (step 950), 
the final demosaiced image can be output from the dig- 
ital signal processor for further processing or display 
(step 955). The final demosaiced image includes the 
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original raw sensor values and either the calculated 
missing color sensor values or interpolated missing 
color sensor values at each pixel location. 
[0058] Although many grid artifacts can be removed 
by using the simple interpolated color sensor values, if 5 
a pixel lies on an edge of an object in the image, using 
the simple interpolated color sensor value may result in 
color aliasing artifacts in the image. In areas surround- 
ing an edge of an object in an image, the neighboring 
color values can vary widely. Therefore, even though 10 
grid artifacts may occur, for those pixels that lie on an 
edge of an object in an image, the calculated missing 
color sensor values may preserve those edges better 
than the simple interpolated color sensor values. 
[0059] Exemplary steps for removing grid artifacts *5 
from an LLR demosaiced image without introducing ex- 
tra color aliasing around edges in an image are shown 
in FIG. 12. Upon receipt of the raw sensor values (step 
1 000) and the division of the raw sensor values into sep- 
arate regions (step 1005), a simple demosaicing algo- 20 
rithm can be applied to the mosaiced image (step 1010) 
to create three full color planes (e.g., R, G, and B), each 
including both original raw and interpolated color sensor 
values. Thereafter, an edge detection algorithm can be 
applied to the image region to determine where the edg- 25 
es in the image region occur (step 1015). The detection 
of edges can be performed using any type of moderately 
effective edge detection algorithm. 
[0060] As an example, the following is a description 
of an edge detection algorithm designed specifically for 30 
the Bayer pattern mosaic. As a first step, the four color 
positions (R, G1, G2, B) in a Bayer pattern are demo- 
saiced separately, as performed in step 1010 above. 
Thus, at each pixel location, there is a red value, a blue 
value, a green value for green pixels next to red pixels 35 
and a green value for green pixels next to blue pixels. 
Thereafter, the G1 and G2 demosaic results are aver- 
aged to obtain the final G results for the interpolated G 
values that are used in subsequent processing. Also, 
the normalized difference of the G1 and G2 demosaic *o 
results are calculated (JG-, - G 2 \(G^ + G 2 )) for each pixel 
to obtain an edge map. 

[0061] In addition to identifying the pixels that lie on 
edges in the image region, the linear regression coeffi- 
cients (slopes and offsets) that describe the linear rela- *s 
tionships between the different color planes in that re- 
gion (step 1020) are determined from both the interpo- 
lated and raw sensor values, as described above in con- 
nection with FIG. 6, or from only the raw sensor values, 
as described above in connection with FIG. 7. From the so 
linear regression coefficients, estimated missing color 
sensor values at each pixel location in that region are 
calculated (step 1025). 

[0062] For each calculated missing color sensor val- 
ue, a comparison is made between the calculated miss- 55 
ing color sensor value and the corresponding interpo- 
lated missing color sensor value to determine a differ- 
ence value (step 1030). If the difference value exceeds 



a threshold amount (step 1035), and the pixel associat- 
ed with the missing color sensor values does not lie on 
edge identified as described above (step 1040), the cal- 
culated missing color sensor value is replaced with the 
interpolated missing color sensor value (step 1045). 
Otherwise, the calculated missing color sensor value is 
used in the final demosaiced image (step 1050). This 
process is repeated for each calculated missing color 
sensor value in the image region (step 1055). 
[0063] Once the estimated missing color values in all 
regions of the image have been calculated (step 1060), 
the final demosaiced image can be output from the dig- 
ital signal processor for further processing or display 
(step 1065). The final demosaiced image includes the 
original raw sensor values and either the calculated 
missing color sensor values or interpolated missing 
color sensor values at each pixel location. 
[0064] In other embodiments, image regions that in- 
clude areas that violate the linear correlation assump- 
tion can be adjusted using different methods. For exam- 
ple, instead of replacing the pixels that violate the linear 
correlation assumption (violation pixels) with the simple 
interpolated color sensor values, a more sophisticated 
linear regression method can be used to determine mul- 
tiple regression coefficients for regions with several dif- 
ferent linear relationships. As a further example, the 
missing color sensor values at violation pixels can be 
replaced with sensor values calculated using a demo- 
saicing method which minimizes color aliasing. By way 
of example, but not limitation, a demosaicing algorithm 
such as described in commonly assigned, co-pending 
U.S. patent application serial number 09/940,825 can 
be used. In another solution, the image can be divided 
into irregularly shaped regions, each containing at most 
two different color regions, thereby removing the cause 
of the linear correlation violation. 
[0065] The innovative concepts described in the 
present application can be modified and varied over a 
wide range of applications. Accordingly, the scope of 
patented subject matter should not be limited to any of 
the specific exemplary teachings discussed, but is in- 
stead defined by the following claims. 



Claims 

1. An image processing system (10) for demosaicing 
an image represented by raw sensor values (30) 
generated by pixels (25) within an array of pixels 
having at least two color planes, each of said pixels 
(25) within said array of pixels producing a respec- 
tive one of said raw sensor values (30) in one of the 
at least two color planes for said image, said image 
processing system (10) being characterized by: 

a buffer (50) for storing at least a group of said 
raw sensor values (30) produced by said pixels 
(25) within a region of said image, said region 
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being less than said image; and 
a processor (40) connected to receive said 
group of raw sensor values (30) stored in said 
buffer (50), determine regression coefficients 
(115) representing the linear relations between 5 
the color planes for said region using at least 
said group of raw sensor values (30) and cal- 
culate missing color sensor values (125) asso- 
ciated with said pixels (25) in said region of said 
image using said regression coefficients (115). 10 

2. The image processing system of Claim 1, wherein 
said regression coefficients (115) comprise a slope 
value and an intercept value representing the linear 
relationship between two of the color planes in said * 5 
region. 

3. The image processing system of Claim 1 or 2, 
wherein said processor (40) is further configured to 
calculate interpolated missing color sensor values 20 
(35) for said region from said group of raw sensor 
values (30), and said interpolated missing color 
sensor values (35) and said group of raw sensor 
values (30) are used to determine said regression 
coefficients (115) for said region. 25 

4. The image processing system of Claim 3, wherein 
said processor (40) is further configured to calculate 
variance sums of squares for each of the color 
planes in said region and covariance sums of 30 
squares between each of the color planes in said 
region using said interpolated missing color sensor 
values (35) and said group of raw sensor values 
(30), and said variance and covariance sums of 
squares are used to determine said regression co- 35 
efficients (115). 

5. The image processing system of one of Claims 1 to 
4, wherein said processor (40) is further configured 

to calculate correlation coefficients between each <o 
of the color planes in said region, said correlation 
coefficients between two of the color planes includ- 
ing a separate correlation coefficient for each po- 
tential position of said pixels (25) in a first one of the 
color planes relative to the position of said pixels 45 
(25) in a second one of the color planes, and said 
correlation coefficients are used to determine said 
regression coefficients (115). 

6. The image processing system of Claim 5, wherein 50 
said processor (40) is further configured to select a 
pixel position for the first color plane associated with 

a maximum one of said correlation coefficients be- 
tween the first and second color planes, and each 
of said raw sensor values (30) in the first color plane 55 
generated from said pixels (25) within said selected 
pixel position relative to said pixels (25) in the sec- 
ond color plane are used to determine said regres- 



sion coefficients (115) between the first and second 
color planes. 

7. The image processing system of Claim 1 or 2, 
wherein said processor (40) is further configured to 
calculate interpolated missing color sensor values 
(35) for said region from said group of raw sensor 
values (30) to produce a first demosaiced image, 
generate a blurred image for said region of said im- 
age through a convolution of said interpolated miss- 
ing color sensor values (35) and said group of raw 
sensor values (30) with a lowpass kernel, and pro- 
duce a difference image by subtracting said blurred 
version from said first demosaiced image, said re- 
gression coefficients (115) being determined using 
said difference image, said processor (40) being 
further configured to produce a second demosaiced 
image using said regression coefficients (115) cal- 
culated from said difference image and output a fi- 
nal demosaiced image for said region by adding 
said blurred image to said second demosaiced im- 
age. 

8. The image processing system of Claim 1 or 2, 
wherein said processor is further configured to cal- 
culate interpolated missing color sensor values (35) 
for said region from said group of raw sensor values 
(30) and compare said interpolated missing color 
sensor values (35) with said associated calculated 
missing color sensor values (125) for each of said 
pixels within said region to detect grid artifacts by 
determining a difference value (145) between said 
interpolated missing color sensor value (35) in a first 
one of the color planes and said associated calcu- 
lated missing color sensor value (125) in the first 
color plane for a select one of said pixels (25) within 
said region, said processor being further configured 
to replace said calculated missing color sensor val- 
ue (125) with said interpolated missing color sensor 
value (35) for the first color plane of said select pixel 
(25) when said difference value (145) exceeds a 
threshold amount (155). 

9. The image processing system of Claim 8, wherein 
said processor (40) is further configured to detect 
edges located within said region of said image, said 
calculated missing color sensor value (125) being 
used regardless of said difference value (145) when 
said select pixel (25) lies on one of said edges. 

10. A method for demosaicing an image represented 
by raw sensor values (30) generated by pixels (25) 
within an array of pixels having at least two color 
planes, each of said pixels (25) within said array of 
pixels producing a respective one of said raw sen- 
sor values (30) in one of the at least two color planes 
for said image, said method being characterized 
by: 
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receiving (500) at least a group of said raw sen- 
sor values (30) produced by said pixels (25) 
within a region of said image, said region being 
less than said image; 

determining (520) regression coefficients (115) 5 
representing the linear relations between the 
color planes for said region using at least a por- 
tion of said group of raw sensor values (30); and 
calculating (530) missing color sensor values 
associated with said pixels (25) in said region 10 
of said image using said regression coefficients 
(115). 
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